Existence and Regularity of a Nonhomogeneous 
Transition Matrix under Measurability Conditions* 

O ; Liuer Ye^, Xianping Guo^, and Onesimo Hernandez-Lerma^'^ 

^ ! ^ The School of Mathematics and Computational Science 

vh . Zhongshan University, Guangzhou, P. R. China 

<^ ; 2£)gpg^^^g^j^g^^Q ^g Matematicas, CINVESTAV-IPN, 

oq : A.Postal 14-740, Mexico D.F. 07000, Mexico 

E-mail: (LY) yeliuer@hotmail.com, (XPG) mcsgxp@mail.sysu.edu.cn, 
^ ' (OHL) ohernand@math.cinvestav.mx 

CIh' 

— January 2007 — 
— Revised: November 2007- 






t:j- . Abstract: This paper is about the existence and regularity of the transition prob- 

"^ ! abihty matrix of a nonhomogeneous continuous-time Markov process with a countable 

^ I state space. A standard approach to prove the existence of such a transition matrix 

QQ ' is to begin with a continuous (in t) and conservative matrix Q{t) = [qij{t)] of non- 

O ■ homogeneous transition rates qij{t), and use it to construct the transition probability 

matrix. Here we obtain the same result except that the qij{t) are only required to 
K> ! satisfy a mild measurability condition, and Q{t) may not be conservative. Moreover, 

H I the resulting transition matrix is shown to be the minimum transition matrix and, in 

addition, a necessary and sufficient condition for it to be regular is obtained. These 
results are crucial in some applications of nonhomogeneous continuous-time Markov 
processes, such as stochastic optimal control problems and stochastic games, which 
motivated this work in the ffist place. 
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1 INTRODUCTION 

Nonhoinogeneous continuous-time Markov processes have applications in a wide variety 
of contexts, including stochastic control problems P, Uni [HI [131 [HI [IHl [20] , stochastic 
games [71 [H [TTl [18] , and queueing systems and stochastic networks [3l [TTl [19] , to name 
a few. As is well known, such a Markov process is uniquely determined by its transition 
probability matrix P{s,t) = [Pij{s,t)] (for i,j G S and < s < t), which in turn is 
usually constructed from a given matrix Q{t) = [qij(t)] of transition rates Qijit), for 
t > 0. Therefore, a natural question is, under what conditions on Q{t) is the transition 
matrix P{s,t) uniquely determined? 

To answer this question, a standard approach — which can be traced back to Feller's 
1940 paper [5] — is to assume that the transition rates qij{t) are continuous in t > 0; 
see, e.g., [11 El [Tl [HI [HI [TOl [131 UHl l2^- This continuity requirement, however, imposes 
severe restrictions in some applications, for instance, in stochastic control and game 
theory, where discontinuous "policies" typically lead to discontinuous transition rates. 
To illustrate this situation, which was the main motivation for this work, let us consider 
the following example. 

Example. Consider a single-server queueing system in which the state variable 
i denotes the number of jobs in the system at each time t > 0. Suppose that a 
controller wants to control the system's service rate /i according to the current state 
i E S := {0, 1, . . .}. When the state is i at some time t, the controller takes a service 
rate /j, from a given finite set A[i) of available "actions" at state i; then the system 
transfers to another state j according to the transition rate induced by the chosen 
fi, and the process is repeated. The choice of service rates is done according to so- 
called control policies it = {rct, t > 0}. If the controller is using a particular policy 
71 = {-Kf, t > 0} and the state at time t is i E S, then the controller takes the service 
rate 7Tt{i) G ^(0- To be more specific, consider the policy vr given by 

oo 

M^) ■= ^f^kii)'i-[k,k+i)it), (1.1) 

fc=0 

with /ifc(i) G A{i) for alH G S* and A; > 0. Hence, when using this policy, if the present 
state is i, then the controller chooses the action fik{i) G A{i) during the time interval 
[k, k + 1). Obviously, Tftii) is measurable in t > 0, but not continuous. Similarly, if q!^- 
denotes the transition rate from i to j under fi G A{i), then the matrix Q'^{t) = [qfjit)] 
of transition rates when using n has elements 



&):=J^q^f%,,^^){t). 



fc=0 

Therefore, the transition rates are measurable in t > but not continuous, and so we 
cannot use the standard Markov chain theory to show the existence of transition prob- 
ability functions P[j{s, t) induced by the discontinuous control policy vr defined in (1.1). 
An analogous situation occurs in stochastic game problems [3 |B|, where discontinu- 
ous "strategies" usually lead to discontinuous transition rates. This is precisely what 
motivated our paper — to establish the existence and regularity of a nonhomogeneous 
transition matrix without requiring the transition rates to be continuous. 



Summarizing, our main objective is to replace the above-mentioned continuity re- 
quirement by a mild measurability condition under which we obtain the existence and 
uniqueness of a transition matrix P{s,t), even if the Q{t) matrix is not conservative. 
(See Section 2 for definitions.) In fact, the existence and uniqueness of P{s,t) under a 
measurability condition have been considered in [15], but the results there are stated 
without proofs and assuming, in addition, that Q{t) is conservative. We also obtain 
some new results (see, for instance. Theorems 2(i) and 2(iii), Theorem 3(ii)). 

In this paper, firstly, we introduce a precise definition of a nonhomogeneous transi- 
tion matrix (Definition 1), which is weaker than previous definitions, e.g., as in [3l[6|[T2]. 
Even in this weaker context, we can obtain some key properties of the transition matrix 
(Theorem 1). Secondly, given a Q{t) matrix satisfying our measurability condition, we 
construct a nonhomogeneous transition matrix (Theorem 2), and, finally, we give a 
necessary and sufficient condition for this transition matrix to be unique and regular 
(Theorem 3). 

The rest of this paper is organized as follows. In Section 2 we present the definitions 
and main results concerning nonhomogeneous transition matrices. The proofs of our 
results are all given in Section 3. In Section 4 we state some conclusions. 

2 MAIN RESULTS 

2.1 Nonhomogeneous Pretransition Matrices 

Throughout the following S denotes a given countable set. 

Definition 1. A real-valued matrix P{s,t) = {Pij{s,t), i,j G S), defined for all 
0<s<t<oo, is called a nonhomogeneous pretransition matrix if it satisfies the 
following for every i,j G S and < s < t < oo: 

(i) 

P,j{s,t) > 0, and ^P,,(s,t) < 1; (2.1) 

(ii) 

P^Jis, t) = J2 ^^'^(^' «)^fcj(^' *) Vs < n < t; (2.2) 



k&S 



m 



lim \Pij{s, s + h) — 6ij\ = uniformly in j G S, and Pij{s, s) = 6ij, (2.3) 

where 6ij stands for the Kronecker symbol {6ij = ii i ^ j ; 6ij = 1 ii i = j). 

If in addition ^,g5-Pij(s, t) = 1 for every i G 5, then P{s,t) is said to be a nonho- 
mogeneous transition probability matrix, and its elements Pij{s,t) are called transition 
functions. 

The equation (2.2) is known as the Chapman-Kolmogorov (C—K) equation. Our 
definition of a pretransition matrix is weaker than that in [3l El [I2], but still we can 
obtain most of the standard results. In particular, the following theorem is essentially 
the same as Theorem 1.1 and Theorem 1.2 in Chapter 3 of \T2\. 



Theorem 1. For any nonhomogeneous pretransition probability matrix P{s,t) we 
have: 

(i) 

Pii{s, t)>0 y ie S, 0<s<t<oo; (2.4) 

(ii) 

\Pij{u,t)-Pij{v,t)\<l-Pii{uAv,u\/v) y i,j e S,0<u,v <t, (2.5) 

where u Av = min('U, v), u\/ v = max(-u, v); 

(iii) Pij{s,t) is continuous in s G [0,t] (right-continuous in 0, left-continuous in t), 
and uniformly in t > and j & S; 

(iv) Pij{s,t) is continuous in t G [s, +cxd) (right-continuous in s), and uniformly in 

jeS; 

(v) The following holds: 

Pn{s,t)-1 Pu{s,t)-1 

hm = hm =: qii{s) < 0, 

t^s+ t — S t-s^O+ t — S 

lini:^M^= lim M^ =:g„(s)>0 for j y^ t; 

t^s+ t — S t-s^O+ t — S 

(vi) For every i E S and s > 0, J2j^i1ij{^) — ^Qui^)- (Hence, %(s) < qi{s) for all 
i,jES and s > 0, where qi{s) := —qii{s) > 0.) 

Proof. See subsection 13. 1[ D 

2.2 Nonhomogeneous Q(t)-Transition Matrices 

In this subsection we introduce the definition of a nonhomogeneous Q{t)-matrix and 
a nonhomogeneous transition matrix induced by Q(t), which are related by our key 
hypothesis, Assumption A. 

Definition 2. For each i,j G S, let %(t) be a real- valued function defined on [0, +oo). 
The matrix function Q{t) = {qij{t),i,j G S) is said to be a nonhomogeneous Q{t)- 
matrix on S if for every i,j & S and t > satisfies that: 

(i) 

< qijit) < oo if i 7^ j, and < —quit) < oo; (2.6) 

(ii) 

J]%W ^0 V^G^. (2.7) 

If in addition ^ %(t) = for alH G 5 and t > 0, then we say that Q(t) is conservative. 



We now introduce our measurability-integrability assumption with respect to the 
Lebesgue measure on [0, +00). 

Assumption A Let Q{t) be a given nonhomogeneous Q(t)-matrix. For every b > 
a > and i,j G S, we have: %(t) is Borel-measurable in t G [a, b] if z 7^ j, and quit) 
is integrable on [a, b]. 

The following definition relates a Q(t)-matrix and a pretransition matrix. (The 
abbreviations a.e. (almost everywhere) and a. a. (almost all) refer to the Lebesgue 
measure.) 

Definition 3. Let Q(t) = {qij(t),i,j & S) (t > 0) be a nonhomogeneous Q(t)- 
matrix satisfying Assumption A. If a nonhomogeneous pretransition matrix P{s,t) = 
{Pij{s,t),i,j E S) {0 < s <t < cxd) satisfies that, for every i,jES and a.a. s > 0, the 
partial derivatives 

(1) -^^ and 

(ii) 

-P,,(s,t)|i=.+ = £m ^^^ ' ^ ^ ^ = q.,{s) (2.8) 

exist, then P{s,t) is called a nonhomogeneous Q{t)-transition matrix. Further, such a 
Q(t)-transition matrix is said to be regular ii P{s,t) is a transition probability matrix 
and it is the unique transition probability matrix that satisfies (12. 8p . We call Pij{s,t) 
a nonhomogeneous Q{t)-transition function or simply a Q{t) -function. 

2.3 Existence of a Nonhomogeneous Q(t)-Transition Matrices 

The question now is, given a nonhomogeneous (5(t)-matrix, how can we ensure the 
existence of a nonhomogeneous (5(t)-transition matrix? The following Theorem 2 shows 
that this can be done if the Q{t)-ma.thx satisfies Assumption A. To obtain a regular 
(5(t)-transition matrix, however, we have to go a bit further and construct a minimum 
(5(t)-transition matrix on which we can readily impose conditions for it to be regular. 
This is done in Theorem 3. 

Theorem 2. Given a nonhomogeneous Q(t)-matrix satisfying Assumption A, let 

dij{u) := 6ij{-qii{u)) W i,j e S, u>0, (2.9) 

and for each i,j G S and < s < t < 00, define recursively 

p(^\s^t) := Sij e^»«''(")'^", (2.10) 

J2 e^^-^'^^'^lq.kiu) + d,k{u)]P^^^{u,t)du Vn > 0. (2.11) 

- k&S 

Let 

00 

P,,(.,t):=^ P;?(^,t). (2.12) 

?i=0 



Similarly, for each i,j G S and < s < t < oo, define recursively 

Q^^\s,t) := 6,, e/>-(")'^", (2.13) 

E Qi\'^^) e/>"('')'^^[g,,(«) + d,jiu)]du yn > 0. (2.14) 

- fees 
Let 

oo 

Q.,{s,t):=Y, Qt\s,t). (2.15) 

?i=0 

Then, for every i,j & S and < s < t < oo, we have 



(i) 



Pjj (s,t) = Qjj (s,t) for all n > 0, and so Pij{s,t) = Q^j{s,t)] (2-16) 



Pij(s, t) = 1^ Pik{s, v)qkj{v)dv + (5i_ 



(2.17) 



111 



Piji^.t) = f 5^gifc(t;)Pfe,(t;,t)rft; + <5i,-; (2.18) 



fees 

(iv) P{s,t) = {Pij{s,t),i,j G S*) is a nonhomogeneous (5(i)-transition matrix. 

Proof. See subsection 13. 2[ D 

The following theorem states our main results in this paper. It shows that, whether 
Q{t) is conservative or not, the (5(t)-transition matrix P{s,t) constructed in Theorem 
2 is minimum; see (I2.19p . Moreover, it gives a reasonably mild necessary and sufficient 
condition for P{s,t) to be regular; see (12.201) . 

Theorem 3. Assume that Q{t) is a nonhomogeneous Q(t)-matrix satisfying Assump- 
tion A, and let P{s,t) be the nonhomogeneous Q(t)-transition matrix in Theorem 2. 
Then: 

(i) P{s,t) is the minimum (5(t)-transition matrix, that is, for any nonhomogeneous 
(5(t)-transition matrix P{s,t) = {Pij{s,t), i,j G S), we have 

Pij{s,t) > Pij{s,t) Wi,j e S, < s < t < oo. (2.19) 

(ii) P{s,t) is regular if and only if 

Y^ / 'YPikis,v)qkjiv)dv = y ie S, 0<s<t<oo. (2.20) 

jes "^^ fees 

Proof. See subsection 13.31 D 



3 PROOFS 

3.1 Proof of Theorem [T] 

To prove Theorem [1] we will use the following lemma in which, ioi h > and an integer 
m > 1, we denote by APi,B{s,s + mh) the probability of transition from the state i 
at time s to the set B at time s + mh while avoiding the set A at times s + kh for 
k = 1, . . . ,m — 1. Observe that 

a-Pj,b(s, s + h) = Pi^sis, s + h), 

and, for m > 2, 

APi,B{s, s + mh) = ^ APir{s, s + {m - l)h)Pr^B{s + (m - l)/i, s + mh). 

Lemma 1. Let P(s, t) = {Pij{s, t), i,j G S") be a nonhomogeneous pretransition matrix 
and, for < s < t, let < /i < t — s, n := [h~^{t — s)], and 1 < m, < n. Then: 

(i) For every i & S and B G S, 

m 
Pi,B{s,t) = ^^APik{s,S + lh)Pk,B{s + lh,t) 
1=1 k&A 

+ ^A-Piyt(s, s + mh)Pk,Bis + mh,t). (3.1) 

(ii) For every < e < 1/3, there exists < S < 1 such that when t — s < S, we have 

n 

Pij{s, t) > (1 - 36) Y, P^jis + {l- l)/i, s + lh) V jV i- (3.2) 

1=1 

Proof, (i) We will use induction on m. For m = 1, (13 .ip holds by the definition of 
APi,B{s, s + m,h) and the C-K equation (12.21) . Now assume that (13.11) holds for m — 1. 
We will prove that it holds for m. Indeed, by the definition of APi,B{s, s + m,h), the 
C-K equation (12.21) again and the induction hypothesis, we have 

PiAs,t) 

m—l 
= YYAPik{s,S + lh)Pk,B{s + lh,t) 

1=1 keA 

+ J2 ^^^^(*' s + ("^ - l)h)Pk,B{s + {m- l)h, t) 



k<^A 
m 



y^ y^ APikis, s + lh)Pk,B{s + Ih, t) - ^ APik{s, s + mh)Pk,B{s + mh, t) 
1=1 fceA feeA 

+ Yl APik{s, s + (m - l)h)Pk,B{s + (m - l)/i, t) 

k^A 

7 



+ ^ APik{s, s + mh)Pk,B{s + mh, i) - ^ APik{s, s + mh)Pk,B{s + mh, t) 

k^A k^A 



y^ y^ APik{s, s + lh)Pk^B{s + Ih, t) + ^ A-Pifc(s, s + mh)Pk,B{s + m/i, t) 
Z=l fceA fc^A 

+ ^ APifc(s, s + (m - l)/i)Pfc,B(s + (m - l)/i, t) 

k(^A 

- ^ APikis, s + mh)Pk^B{s + m/i, t) 



kas 



= ^ ^ A^ifcls, s + lh)Pk,B{s + Ih, t) + ^ A^ifcls, s + mh)Pk,B{s + m/i, t). 

1=1 k&A k^A 

This completes the induction. 

(ii) Taking B = A = {j} and m = n in (13.11) . we obtain 

n 

Pij{s,t) = y^jPij{s, s + lh)Pjj{s + lh,t) + y^jPikis, s + nh)Pkj{s + nh,t). (3.3) 

1=1 k^j 

On the other hand, (12.31) imphes that for any given < £ < 1/3 and j ^ i, there 
exists < (5 < 1 such that when < t — s < 6, we have Pu^s.t) > 1 — e, Pjj{s,t) > 
I — e, and Pij{s, t) < e. These facts together with (13. 3p imply that ioi h < t — s < 5 
and i ^ i 

e>l-Pu{s,t) > 5^P.fe(s,t)>Pi,(s,t) 

kj^i 
n 

- X] jPiji^^ ^ + ^h)Pjj{s + ^h, t) 
1=1 

n 

1=1 

so 

n 

e 



J2^P,^{s,s + lh)<- 



1=1 
Note that, for each 1 < I < n, 

i-i 
Pii{s, s + Ih) = jPii{s, s + Ih) + y^ jPiji^-, s + rah)Pji{s + m/i, s + Ih). 

m=l 

Thus 

i-i 

jPii{s,s + Ih) = Pii{s, s + lh) - y^ jPjjjs, s + mh)Pji{s + m/i, s + //i) 

m=l 
Z-1 



> Pii(s, s + //i) - ^ jPijis, s + mh) 

> l-£- 



m=l 



1-e' 



and so 

n 

Pijis, t) > ^ jP»i(s, s + lh)Pjj{s + Ih, t) 
1=1 

n 

= ^^jPiris, s + {l- l)h)Prj{s + (/ - l)h, S + lh)Pjj{s + Ih, t) 

1=1 r^j 
n 

> ^jPii{s,s + {I - l)h)Pij{s + (/ - l)h,s + lh)Pjj{s + lh,t) 
1=1 

n 

> {l-e- ^^) Y, Pijis + (/ - l)h, s + lh){l - e) 

1=1 

n 

> {l-3e)YPij{s + {l-l)h,s + lh). 



1=1 



This completes the verification of (3.2) and also the proof of the lemma. D 

With Lemma [H we can easily prove Theorem [H 
Proof of Theorem Ui 

(i) By the C-K equation (12. 2p . we obtain 



n , 

Pn(s,t)>n^-(^+ 

fc=l ^ 



(t-s),s + -(t-s) ) Vra > 1. 



n n 



Hence, for n sufficiently large, (12. 4p follows from (12. 3p . 
(ii) Let < u < t; < t < oo. By the C-K equation ([22D 

P,j(M,t) - Pii{v,t) =J2P^k{u,v)Pkj{v,t) + {Pu{u,v) - l)Pij{v,t). 

Applying (12. ip . we obtain 

Pijiu,t) - P^Jiv,t) > {Pu{u,v) - l)Pij{v,t) > Puiu,v) - 1, 

Pij{u, t) - Pij{v, t) <Y Pik{u, v)Pkj{v, t) < ^ Pik{u, v) <1- Pii{u, v). 

k^i k^i 

These inequalities yield (12. 5p . 

(iii) Using (12. 5p and (12. Sp . for each /i > we have 

\Pij{s + h,t)- Pij{s,t)\<l- Pii{s,s + h) -^0 as /i -^ 0+, 

\Pijis,t) - Pij{s -h,t)\<l- Pii{s -h,s) -^0 as h^ 0+, 

which together with (12. 3p yield (iii). 

(iv) By the C-K equation (12. 2p . for each h > we have 

Pij{s, t + h)- Pij{s, t) = J2 Pik{s, t)Pkj{t, t + h)-{l- Pjj{t, t + h))Pij{s, t). 

k^3 

9 



Therefore 

kes 
and 

Pi,{s,t+h)-Pi,{s,t)<j2Pik{s,t)Pkj{t,t+h)<j2Pik{s,t){i-Pkk{t,t+h)). 

kj^j kes 

Hence, as /i — > 0"*", 

\Pij{s,t + h) - Pi^{s,t)\ <J2Pik{s,t){l - Pkk{t,t + h)) -^ 0, 

kes 

by (12.31) and the Dominated Convergence Theorem 0. It follows that Pij{s,t) is right- 
continuous in t G [s, +cxd), uniformly in j G S. 

Similar arguments show that Pij{s,t) is left-continuous in t G [s,-|-oo), uniformly 
in j G S. 

(v) To avoid trivial situations we suppose that i G S" is not an absorbing state, 
i.e., Pii{s,t) ^ 1. For < s < t < +cx), let f{s,t) := — logPjj(s, t), which is a 
well-defined function, nonnegative and finite. Since Pii{s,t) > Pii{s,u)Pii{u,t), for 
0<s<M<t<cxD, we have f{s,t) < f{s,u) + f{u,t). Now for each s > 0, let 
qi{s) := sup ^^^i^. We will next prove that the limit of ^^ exists and equals qi{s). 

s<t<oo 

Obviously, by definition of qi{s), 

hmsup < qi[s). 

t-s^O+ t — S 



t-s 

take n such that t — s = nh + e, with < s < h. Then 



Therefore, it is sufficient to argue that liminf ;^' ' > qi(s). Given any < h < t — s, 



f{s,t) ^ nh f{s,s + nh) f{s + nh,t) 
t — s ~ t — s nh t — s 

Now take the limit of both sides as h ^ 0~^, and note that n/i — i> t — s as e — *> 0+, so 
that the continuity of Pij{s,t) implies that /(s + nh,t) = f(t — e,t) -^ 0. Hence we 
have 

t — s h^o+ nh t-s^o+ t — s 

and so 

lim^- = qi{s). 

t-s^O+ t — S 

Finally, recalling the definition of f{s,t), we have 

1-P,,(g,t) 1 - e-/(--*) l-e-/(-.*)/(g,t) 
iim = hm = hm — — = qi[s). 

t-s^O+ t — S t-s->0+ t — S t-s^O+ /(s,t) t — S 

This proves the first part of (v). Next we prove the second part. 

10 



To this end, first note that (12. 3p imphes that for any given < e < 1/3 and 
j 7^ i, there exists < 6 < 1 such that when < t — s < 6, we have Pii{s,t) > 
1 — e, Pjj{s,t) > 1 — e, and Pij{s,t) < e. Since (12 .Sp holds uniformly in j G S, for 
0</i<t-s, Acs, and i ^ A, it follows from ( 13^ that 

n 

Pi,A(s, t) > (1 - Be) Y, Aa(s + (/ - 1)/^, s + //i). (3.4) 

1=1 

In particular, taking A = S — {i,j} in (13. 4p . we obtain 

n 

Pi,s-{i,ds, t) > (1 - 3£) J2 P^,s-{i,j}{s + (/ - l)/i, s + //i). (3.5) 

1=1 

On the other hand, taking B = A = S — {i} and m = n in (13. ip . it follows that 

n 
^i,5-{i}('5>^) = XI XI APik{s,s + lh)Pk,s-{i}{s + lh,t) 
1=1 kes-{i} 

+ APii{s, s + nh)Pi^s~{i}{s + nh, t) 

n 

< X X APik{s,s + lh)+ APii{s,s + nh)Pi^s-{i}{s + nh,t) 

1=1 k£S-{i} 

n 

= X APi,s~-{i}is, s + lh) + APii{s, s + nh)Pi,s-{i]{s + n/i, t) 
1=1 

n 

= J2 APii{s, s + {l- l)h)Pi^s-{i}{s + (/ - l)/l, s + Ih) 
1=1 



+ APii{s,s + nh)Pi^s-{i){s + nh,t) (since A = S - {i}); 



consequently, 



Pi^s-{i}{s,t) < J2Pi^s-{i}{s + {l-l)h,s + lh) 
1=1 
+ APii{s, s + nh)Pi^s-{i}{s + nh, t). (3.6) 

Subtracting (13.50 from (13. 6p . and using (13.50 again, we obtain 

n n 

Pij{s,t) < J]Pij(s + (/-l)/l,S + //l)+3£^Pi,5_|ij}(s + (/-l)/l,S + //l) 

1=1 1=1 

+ APii{s, s + nh)Pi^s-{i}{s + nh, t) 

n „ 

< Y^ P,j{s + {l~l)h,s + Ih) + --^P^s_^,.^{s, t) 
1=1 
+ APii{s, s + nh)Pi^s~{i}{s + nh, t). (3.7) 
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Recalling that e was arbitrary, taking the limit of both sides of (13.71) as £ — i> 0, we see 
that 

n 

Pij{s, t)<J2 Piji^ + (^ - 1)^' s + lh)+ APii{s, s + nh)Pi^s~{i}{s + nh, t). (3.8) 
1=1 

Summarizing, by (13. 2p 

(1 3c) V ^'^^^ ^^^~ ^^^' ^ + ^^^ < -^^j(^''^) (3 9) 

■^ n/i ~ nh ^ 



1=1 



whereas by (13. 8p 

Pijjs^t) ^^^ Pij{s + {l-l)h,s + lh) APiiis,s + nh)Pi^s-{i}is + nh,t) 

nh ~ ^-^ nh nh 

1=1 

To conclude, note that n/i ^ t — s as /i — > 0"*". Hence, using (12. 3p . (13. 9p . and (I3.10p 
we obtain 

^ p,,(g + (/-i);i,5 + //i) p.,(s,t) 

limsup > — ^^-^^ ^ r^ = limsup — . (3.11) 

From this equality and (13.90 it follows that 

,. P^3{s,t) 1 P,,(s,t) 

limsup — < — -. 

t-s^O+ t — S 1 — 35 t — s 

Hence, taking the limit infimum of both sides we obtain 

limsup — < limmt — . 

t_s^O+ t — S 1 — 2)6 t~s—>0+ t — S 

Finally, letting e ^ we conclude the proof of part (v) . 

(vi) By (2.1), Pii{s,t) + Y.j^iPiiis^'^) < 1 for all i,j e S and t > s > 0, or, 
equivalently, J2j=tiPij{^^^) < 1 ~ Pii{s,t). Hence (vi) follows from (v) and Fatous' 
Lemma. D 



3.2 Proof of Theorem H 

Proof, (i) By Assumption A and the definition of qij(t), the functions P^j (s,t) and 
Qj • (s, t) are well defined for every n > 0. 

To prove (I2.16p . we use induction on n > 0. Obviously, P^j (s,t) = Q^j (s, t) and 
p'-^\s,t) = Q^i^\s,t), by (1211]) and (l27Hp . Now assume that fl216D holds for some 
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n > 0; we will show that it holds for n + 1. By the induction hypothesis and (12.111) 
and fl2.14p . we have 



n''\s,t) 



J2 e^^''^^^^^''^[q,k{u) + d,k{u)]Q^^{u,t)du 
J2 e/"^-(^)'^''[to(n) + d,,(M)]- 



kes 
t 



J2 Qki '\u,x) e^^'^^^^'^^'^[qi,{x) + dij{x)]dx 



" les 



du 



J2 e/>^-^(^)'^^[g,,-(x) + rf,,(x)] 



les 



t(«-i), 



Y^ e^^'i^^^^^''^[q,k{u) + d,k{u)]Qir"{u,x)du 

k&S 

J2 e^^'^^^^^^'^^iqi^ix) + di,{x)]P^;'\s,x)dx 



dx 



les 



Y^ Quis.x) e/^*^(^)'^ng;,(x) +d,,(x)]da; 



les 



Qii is,t), 



Consequently, fl2.16p holds for n + 1. This completes the induction and verifies (i). 

(ii) To prove fl2.17p . we first prove, by induction on n, the following statement: for 
every n > and t > s > 0, we have 

fj2P'^:k^'''('^''^^^^(''^^''=fT.Pi~'('^''^M^^ (3.12) 

•^^ keS "^^ k&S 

Indeed, by f l2.10p and (12.91) . for n = a direct calculation gives 



Yl ^*fc (^' v)dkj{v)dv = 6ij - Pif{s, t), 

- kes 



and so fl2TT]) together with I^AQh and (El) gives 



Y^ik {s,v)dkj{v)di 



kes 



[ J2 [YI e^"""^"^'" Mu) + duiu)]5ik ei'""''"(^)'^^ 5kji-qkkiv))dudv 
-^^ kes -^^ i&s 

I Hf e^s^^^i-)'^- [q,k{u) + d,k{u)]6kj ef^'^'''^^^''%-qkk{v))dudv 
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■"^ fees 
t 



o!^'ikk{x)dx/_ 



J2 ei'"^-(^)'^^ hk{u) + d,k{u)]6kj 
fees 

fees 



qkk{v))dv 



du 



kes 
t 

^ kes 
-M 



■(0), 



hkiu) +dik{u)] P^j {u,t)du 



Yl Pik (s. u)\qkj{u) + dkj{u)]du - P^^ (s, t) 



kes 



Hence, (13.121) holds for n = 0. 

Now assume that (13.121) holds for some n > 0; we will show that it holds for n + 1. 
By the induction hypothesis and (12.111) . we obtain 

YP^i^^^\s,v)dkj{v)dv 

- kes 

■^^ fees L * les 



dkj{v)dv 



J2 e^>-^-)^^ [qu{u) + du{u)] 
les 



^{"+1), 



YP\r^'{%'v)dkj{v)dv 
" fees 



du 



les 

E ^ik (w, ^)[4i(t^) + qkj{v)]dv - Pi]'^^ (m, t) 

- fees 

J2 Yl e^"'"^"^'" [g.z(M) +rf,Kw)] PL"\«,^)[4,(t^) + ?fei(^)] ^^^w 



^ kes ^ "■ les 
t 



- I Y. e^^^-^^)''^ [qa{u) + da{u)] P^i;-''\u,t)du 
•^ ■'' les 

[Y [Y e^"^"(^')'^ [qa{u) + da{u)] T 
"^■^ fees L"^'^ les 



Ik [u,v)du 



[dkj{v) + qkj{v)]dv - Pj (s, t) 



Consequently, (I3.12p holds for n + 1 and this completes the induction. 
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Now note that (13.121) gives 



„i 00 



n=0 "''' ike5n=0 

This equahty and (2.12) yield 

ft 



k€Sn=0 



r"(M) 



n=0 "^^ k€S n=0 

= ^ij+ ^Pik{s,v){dkj{v) + qkj{v))dv - ^Pik{s,v)dkj{v)di 

■^^ k€S •'^ k€S 

y^ -Pifc(s, v)qkj{v)dv + 5ij, 



fce5 



which proves (12.171) . 

(iii) The proof of (I2.18P is quite similar to that of (12.171) . We first prove, by induction 
on n, the following statement, which is analogous to (3.12): for every n > and t > 
s > 0, 

/ ^dik{v)Qi^'^ {v,t)dv= ^[q^k{v) + d^k{v)]Q^j {v,t)dv-Qi'^~^ {s,t). (3.13) 

■^^ keS "^ ^ k&S 

Indeed, by (I2.13p . for n = we obtain 

- kes 
This fact and (127[4D yield 

^dik{v)Q,^j {v,t)dv 
kes 

Y,5a{-qu{v)) e/""''-(^)'^^e/^*^(^)'^^[g,,(M) + rf;,-(M)] dudv 



kes 
t ft 



dv 



S JV i^g 

l€S 



i-quiv)) e^"^''(")'^"rft; 



^i:ini-)'i-[q^^(^u) + di,{u)]du 



f 5]]**K1 - e^"««(^)^^) e^"«^^(^')'^^[gzj(M) + dij{u)] du 



J^feH + 4Kw)] -^^i ei'>«(^)'^^ ciu 



leS 
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n^^^Q nApfllni'^)'^' 






''[qij{u) + dij{u)] du 



7t(1), 



Y,[qu{u) + du{u)]Q'';;\u,t)du-Qy{s,t). 



les 



Hence, (13.131) holds for ri = 0. 

Now assume that (13.131) holds for some n > 0; we will show that it holds for n + 1. 
By the induction hypothesis and (12.141) . we obtain 



^dik{v)Qkj iv,t)dv 



keS 



y^^dikjv) 

k£S 



!"+'V„ ,,v/^-(-)'^- 



J2Qkr'{v,u)eJ^''''''^'^[qM + diAu)] du 



l£S 



dv 



/ ^ / J2^ik{v)QM^^\v,u) dv 

Js i^g Js ,,.^e 

E 



fcGS 



3/^..W'^^[g;^.(^)+ci^^.(^)] du 



les 



(n+l), 



^hkiv) + dikiv)]Qi^i {v, u) dv - Q^j (s, u) 



k£S 



Jllni^)d^[q^^^u) + dij{u)] du 



E 

i&s 



^hkiv) +dikiv)]Qki {v,u) dv 



k£S 



j:ini-)d^[q^^^u) + di,{u)]du 






^[qik{v) + dik{v)] 



kes 



J2Qm\v,u) e^^^^^^^^'^[qi,{u) + di,{u)]du 



" i&s 



dv 



-Qii {s,t) 



= fY.^q.k{v) + d,k{v)]Q%^'\v,t) dv-Q^;'-'\s,t). 
•'^ kes 

Consequently, (I3.13P holds for ra + 1 and this completes the induction. 
Now note that (13.131) gives 

CX3 pi OO „^ OO 
n J S L.^a n Js u^o . n 



kj {v,t)dv. 



n=0 •"■ k&S 

Therefore, by (12.151) . 



fcG5 n=0 



OO «^ OO 

n=0 "^^ fceS n=0 
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k&S 



Sij+ ^[qikiv) + d^kiv)]Q^j{v,t)dv- I ^dikiv)Qkj{v,t)di 

•^'^ k&S 

X^ (lik{v)Qkj{v, t)dv + 5ij. 



kes 



This equality and (2.16)_give (EH]). 

(iv) To prove that P{s,t) is a nonhomogeneous (5(t)-transition matrix we need to 
show that Pij{s,t) satisfies fl2.ip - fl2.3p . and that the partial derivatives in (i)-(ii) of 
Definition 3 exist. 



T5(0) 



To prove (2.1) we already know that P^- {s,t) > 0, by (12.101) . Suppose now that 



^(n) 



Pij ("S, t) > for some n. To prove that this holds for n + 1, we use (12.111) and (12.61) 



to obtain 



?r "(-.*) 



E 

fcg5 

E 

fees 



ols liii'")'^'" 



oI^QiiMdv 



in), 



[qik{u) + dik{u)\Pf,. {u,t)du 



N75(") 



q,k{u)Plj'{u,t)du>0 [hj^M)- 



Hence Pij{s,t) > 0. 

To prove that Pij{s,t) satisfies the second part of (12. ip . it suffices to show that 



5^p!;\.,t)<l Vn>0, 
because then in a similar manner we can prove show that 

N 
jg5 n=0 

Therefore, since 

oo N 



(3.14) 



(3.15) 



n=0 n=0 

(12. ip follows. Now, to prove (I3.14p . we use induction on n. For n = 0, (3.14) trivially 
holds, by fl2A0|) . 

Suppose now that (I3.14p holds for some n, that is. 

To see that this holds for n + 1, we use (12. lip and monotone convergence to obtain 



E7'r"(M) 



J2 ei'"^-^^)* hk{u) + d,k{u)] 



fcG5 



En7(".*) 



jes 



du 



"^* k&S •^'^ k&S 



< I e 

's 



k&S 
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Hence, by (EIZD and ([23]), 



jeS "' * k€S 



which yields fl3.14p . 

Now we will verify that Pij{s,t) satisfies the C-K equation (\2.2\j . Observe that this 
holds if and only if, for every n > and s < u < t, 



p^\s^^ = E E^- '(^'-)^ir'(-'^)- (3-16) 



We will prove (3.16) by induction. In fact, for n = (13.161) follows from (I2.10p . 
Suppose now that (I3.16P holds for some n > 0. To prove (I3.16P for n + 1, we use the 
induction hypothesis and (12. lip , to obtain, for any s < r <t, 






k€S 

n 

(m) , ■ -j^{n—m) 



kes 

Y e^s^-^^)<i^ [q,,{u) + d,k{u)]Pj;fiu,t)du 
fees 

n 



l^l^Pki iu,r)Pij (r,t) 



du 



where 



k&S 



Since (13.161) holds for n = 0, recalling (2.10) we then have 

les 
and so 

n+l 

m=0 les 

Hence, (13.160 holds for all n, and, as already noted, (12.20 follows for Pij{s,t). 

To see that Pij{s,t) satisfies (Q, using (|2:9|) - (I2A2D . (|23|) . (12^1) and (l3J[5|) . we 
obtain 



r^jyS, tj Oij I 

oo 

>75(n+l), 

ra=0 



< (i_ e/^*«"(^)'^^)+x:?;r\^,t) 



i.^a n 



tu 

fce5 n=0 






'u)\du 



■^ fce5 



< (1_ e/. 9»'(^)'^^ )+ / e^"'?"'(^)'^'' [-qii{u)\du 
-^ as t ^ s"*". 

This implies the desired resuh. 

To summarize, we have just shown that P{s,t) = {Pij{s,t), i,j G S) is a non- 
homogeneous pretransition matrix. Therefore, to complete the proof that it is a non- 
homogeneous (5(t)-transition matrix, it only remains to verify that P{s,t) satisfies (i) 
and (ii) in Definition 3. But in fact from (12.1 71) . (12.181) and the fundamental theorem of 
calculus for Lebesgue integrals we can obtain a bit more than (i), namely, Kolmogorov's 
forward and backward equations 



kes 



kes 

for a.a. t > s > 0. Moreover, if we take t = s in the forward equation, we obtain (2.8). 
This verifies (iv) and it also completes the proof of Theorem 2. D 

3.3 Proof of Theorem [3] 

To prove Theorem [3] we need the following facts. 

Lemma 2. Assume that Q{t) is a nonhomogeneous Q(t)-matrix satisfying Assumption 
A. Then for any nonhomogeneous Q(t)-transition matrix P{s,t) = {Pij{s,t), i,j G S) 
we have, for every i,jES and < s < t < cxd, 

(i) 

^^^<-E^^'^(^)^'^^(^'^)' (3-17) 

kes 

(ii) 

P^As^t) > I Y. e^"'"^'^'''' hk{u) + d,u{u)]PkAu,t)du + 5^, e^s'}^du)du ^g^^g^ 
"^'* kes 

Proof, (i) To prove fl3.17p . we use the C-K equation (12. 2p to obtain 

-[P^jis + h,t)-P,jis,t)] 
= -[1 - Puis, s + h)]Pi,{s + h, t) -^ -Pikis, s + h)Pkj{s + h,t). (3.19) 



h 

keS 
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Hence, by Fatou's Lemma, (12.81) and Theorem [T](iii) we have 

hminf^-P,fc(s,s + /i)Pfe,(s + /i,t) > 5^fe(s)Pfe,(s,t). (3.20) 



h^0+ ^-^ h 

fees fees 



Then (i) follows from Definition 3, (Km . and (K^ . 
(ii) By (IXTTj) . we have 



J2 e/"^-(^)'^^ [q,kiu) + d,k{u)]Pkj{u,t)du 

J2 e^^^^^^^^''^q,k{u)Pkj{u,t)du + J J2 e^^"^'''^''^ dik{u)Pkj{u,t)du 






k£S "" k€S 

< - f e^"^-^'')"^ ^^^^du+ r e/"^-('')'^^ (-fc(u))P,,(u,t)du 

This yields (13.181) . and completes the proof of Lemma 2. D 

With Lemma O we can easily prove Theorem [3l 
Proof of Theorem 0. 

(i) Let Pjj (s,t) and Pij{s,t) be as defined in Theorem [2], and let Pij{s,t) be as in 
Lemma [21 We will prove (2.19) by showing that (3.21), below, holds for all n > 0. 
By Lemma [21 we know that Pij{s,t) satisfies (I3.17p . which for i = j becomes 

dP-(u t) 

g < -Y^qik{u)PH{u,t) M leS, 0<M<t<oo. 

" fees 

Since qij{u) > for z 7^ j, we have ^^^ < -qii{u)Pii{u,t), and so -^^^ > 
qii{u)du. Integrating from s to t on both sides and using ( ]2.10p . we obtain Pii{s,t) > 
Pjf (s,t). Hence 

P.,(s,t)>p;°\s,t) Wz,jeS. 

Suppose now that for some n >0, 

n 

p.,(.,t)> j]pl;\.,t). (3.21) 

m=0 

To prove that this holds for n + 1, we use (13.181) and the induction hypothesis to obtain 



Pij{s,t) > 6ije^s'i^du)du + f J2 e/"''-^^)'^^ [q,k{u) + d,k{u)]Pk,{u,t)du 

-^^ k&S 

> 5,, e/>-(")'^" +E / E e^"'"^'^''' [qik{u) + d,k{u)]Pt;\u,t)d 
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r(m+l) 



Pi/is,t) + J2pT ('^^) (by (2-10), (2.11)) 



m=0 



n+1 



En"'(M)- 



m=0 



Therefore, (13.211) holds for all n > 0, and letting n — ;> oo in f l3.2ip . we obtain f l2.19p . 

(ii) By Definition 3 and fl2.17p . we know that P{s,t) = {Pij{s,t), i,j G S) is 
regular if and only if 



iG5 



^ Pikis, v)qkj{v)dv + 5i 



kes 



= 1. 



that is 



E 

iG5 



^Pife(s,i;)gfcj(t^)rft 



fees 



= 0, 



which is the same as (2.20). This completes the proof of Theorem 3. 
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4 CONCLUSIONS 

In this paper we have presented a fairly detailed, self-contained, exposition of the 
construction of a (5(t)-transition matrix starting from a nonhomogeneous Q(t)-matrix 
that satisfies a very mild measurability condition. Moreover, such a transition matrix 
is in fact the minimum Q(t)-transition matrix and we have presented a necessary and 
sufficient condition for it to be unique and regular. In short, this paper efficiently gen- 
eralizes the main results of a nonhomogeneous (5(t)-transition matrix with continuous 
and conservative transition rates qij{t), to the case in which the qij{t) are measurable 
and may not be conservative. 

REFERENCES 

[1] Anderson, W.J. (1991). Continuous Time Markov Chains. Springer, New York. 

[2] Ash, R.B. (2000). Probability and Measure Theory, Second Edition. Academic, 
London. 

[3] Breuer, B. (2003). From Markov Jump Processes to Spatial Queues. Kluwer, Dor- 
drecht. 

[4] Chang, H.S. (2006). Perfect information two-person zero-sum markov games with 
imprecise transition probabilities. Math. Meth. Oper. Res. 64, 335-351. 

[5] Feller, W. (1940). On the integro-differential equations of purely discontinuous 
Markoff processes. Trans. Amer. Math. Soc. 48, 488-515. 



21 



[6] Gikhman, LI. and Skorokhod, A.V. (1996). Introduction to the Theory of Random 
Processes. Dover, Mineola, NY. (This is a reprint of the 1969 edition pubhshed by 
Saunders, Philadelphia, PA.). 

[7] Guo, X.P. and Hernandez- Lerma, O. (2005). Nonzero-sum games for continuous- 
time Markov chains with unbounded discounted payoffs. J. Appl. Probab. 42, 
303-320. 

[8] Guo, X.P. and Hernandez-Lerma, O. (2003). Zero-sum games for continuous- 
time Markov chains with unbounded transition and average payoff rates. J. Appl. 
Probab. 40, 327-345. 

[9] Guo, X.P. and Hernandez-Lerma, O. (2003). Drift and monotonicity conditions for 
continuous-time controlled Markov chains with an average criterion. IEEE Trans. 
Autom. Control 48, 236-245. 

[10] Guo, X.P. and Hernandez-Lerma, O. (2003). Continuous-time controlled Markov 
chains. Ann. Appl. Prob. 13, 363-388. 

[11] Guo, X.P. and Zhu, W.P. (2002). Denumerable-state continuous-time Markov de- 
cision processes with unbounded transition and reward rates under the discounted 
criterion. J. Appl. Probab. 39, 233-250. 

[12] Hu, D.H. (1983). Markov Process Theory with Countable State. Wuhan University 
Press, Wuhan. (In Chinese.) 

[13] Hu, Q.Y. (1996). Nonstationary continuous time Markov decision processes with 
the expected total reward criterion. Optimization 36, 181-189. 

[14] Kakumanu P. (1975). Continuous time Markovian decision process with average 
return criterion. J. Math. Anal. Appl. 52, 173-188. 

[15] Liu, J.Y., Hu, Q.Y. and Wang, J.M. (2004). The foundational assumption of 
continuous-time Markov decision process. Acta Math. Appl. Sinica 27, 756-759. 
(In Chinese.) 

[16] Puterman, M.L. (1994). Markov Decision Processes. Wiley, New York. 

[17] Sennott, L.I. (1999). Stochastic Dynamic Programming and the Control of Queue- 
ing Systems. Wiley, New York. 

[18] Sinha, S. and Jana, S. (2004). Semi- infinite semi-Markov stochastic games. 
Opsearch 41, 278-290. 

[19] Stidham, S. and Weber, R. (1993). A survey of Markov decision models for the 
control of networks of queues. Queueing Systems 13, 291-314. 

[20] Wu, C.B. (1997). Continuous time Markov decision process with unbounded re- 
ward and nonuniformly bounded transtion rates under discounted criterion. Acta 
Math. Appl. Simca 20, 196-208. (In Chinese.) 



22 



